Review

The concept of likelihood and maximum likelihood estimation (MLE) have been at the core of much of statistical modeling for about 100 years

• In 21st Century, likelihood and MLE ideas continue to be foundational

  • The principles we can easily visualize in a few dimensions apply in high dimenstions
    
  • An understanding of these principles helps you understand modern large scale methods

• Understanding the concept of likelihood and the use of MLE methods is key to understanding many parametric statistical methods
  

• Likelihood is a measure of how well a model fits data
  

• MLE is a generic methods for parameter estimation

• MLE used widely for machine learning models, including some deep learning models

Review

Statistical inference seeks to characterize the uncertainty in statistical point estimates

• Statistics are estimates of population parameters

• Inferences using statistics must consider the uncertainty in the estimates

• Confidence intervals quantify uncertainty in statistical estimates

  • Two-sided confidence intervals: express confidence that a value is within some range around the point estimate
    
  • One-sided confidence intervals: express confidence that the point estimate is greater or less than some range of values

Review

Nonparametric bootstrap estimation is widely useful and requires minimal assumption

• Bootstrap distribution is comprised of values of the statistic computed from bootstrap resamples of the original observations (data sample)

• Computing bootstrap distribution requires no assumptions about population distribution!

  • Bootstrap resampling substitutes computer power for paper and pencil statistician power

• Bootstrap resampling estimates the bootstrap distribution of a statistic

  • Compute mostly likely point estimate of the statistic, or bootstrap estimate
    
  • The bootstrap confidence interval is computed from the bootstrap distribution

Review

There are several variations of the basic nonparametric bootstrap algorithm

• One sample bootstrap
  • Inference on single statistic
• Two sample bootstrap
  • Inference on different statistic
• Special cases
  • Correlation coefficients - part of your assignment

Review

Re-sampling methods are general and powerful but, there is no magic involved! There are pitfalls!

• If a sample is biased, the re-sampled statistic estimate based on that sample will be biased
  • Results can be no better than the sample you start with
    
  • Example; the bootstrap estimate of mean is the unbiased sample estimate, \(\bar{x}\), not the population parameter, \(\mu\)
• The sample variance and Cis can be no better than the sample distribution allows
  • Be suspicious of overly optimistic confidence intervals
    
  • CIs can be optimistically biased
• Are computationally intensive, but often highly parallelizable

Introduction Baysian Models

Despite the long history, Bayesian models have not been used extensively until recently

• Two traditions in statistics
  • Frequentist we have been working with previously
    
  • Bayesian statistics
• Limited use is a result of several difficulties
  • Rarely taught for much of the 20th Century
    
  • The need to specify a prior distribution has proved a formidable intellectual obstacle
    
  • Modern Bayesian methods are often computationally intensive and have become practical only with cheap computing
• Recent emergence of improved software and algorithms has resulted in wide and practical access to Bayesian methods

Introduction

Bayesian analysis is a contrast to frequentist methods

• The objective of Bayesian analysis is to compute a posterior distribution
  

• Contrasts with frequentist statistics is to compute a point estimate and confidence interval from a sample

• Bayesian models allow expressing prior information in the form of a prior distribution
  

• Selection of prior distributions can be performed in a number of ways

• The posterior distribution is said to quantify our current belief
  

• Update beliefs based on additional data or evidence
  

• A critical difference with frequentist models which must be computed from a complete sample

• Inference can be performed on the posterior distribution by finding the maximum a postiori (MAP) value and a credible interval

Bayesian Model Use Case

Bayesian methods made global headlines with the successful location of the missing Air France Flight 447

• Aircraft had disappeared in little traveled area of the South Atlantic Ocean

• Conventional location methods had failed to locate the wreckage; potential search area too large
  

• Bayesian methods rapidly narrowed the prospective search area

  • Used ‘prior information’ on aircraft heading and time of sattelite transmisison

Posterior distribution of locations of Air France 447

Bayesian Model Use Case

Kratzke, Stone and Frost developed an optimal search missing planner using Baysian model

• Search areas concentrate on high posterior probability regions
  
• Model accounts for current, wind, etc.

Screen shot from USCG search planner

Bayesian vs. Frequentist Views

With greater computational power and general acceptance, Bayes methods are now widely used

• Among pragmatists

  • Some problems are better handled by frequentist methods
    
  • Some problems with Bayesian methods

• Bayes models allow us to express prior information

• Models that fall between these extremes are also in common use

  • Methods include the so-called empirical Bayes methods.

Bayesian vs. Frequentist Views

Can compare the contrasting frequentist and Bayesian approaches

Comparison of frequentist and Bayes methods

Review of Bayes Theorem

Bayes’ Theorem is fundamental to Bayesian data analysis.

• Start with:

\[P(A \cap B) = P(A|B) P(B) \]

We can also write:

\[P(A \cap B) = P(B|A) P(A) \]

Eliminating \(P(A \cap B):\)

\[ P(B)P(A|B) = P(A)P(B|A)\]

And finally, Bayes theorem!

\[P(A|B) = \frac{P(B|A)P(A)}{P(B)}\]

Bayes Theorem

Bayes Theorem!

Marginal Distributions

In many cases we are interested in the marginal distribution

• Example, it is often the case that only one or a few parameters of a joint distribution will be of interest
  • In other words, we are interested in the marginal distribution of these parameters
    
  • The denominator of Bayes theorem, \(P(data)\), can be computed as a marginal distribution
• Consider a multivariate probability density function with \(n\) variables, \(p(\theta_1, \theta_2, \ldots, \theta_n)\)
  • Marginal distribution is the distribution of one variable with the others integrated out.
    
  • Integrate over all other variables \(\{ \theta_2, \ldots, \theta_n \}\) the result is the marginal distribution, \(p(\theta_1)\):

\[p(\theta_1) = \int_{\theta_2, \ldots, \theta_n} p(\theta_1, \theta_2, \ldots, \theta_n)\ d\theta2, \ldots, d \theta_n\] - But computing this integral is not easy!

Marginal Distributions

• For discrete distributions compute the marginal by summation

• Or, for discrete samples of a continuous distribution

• Example, need to know (un-normalized) posterior distribution of parameter \(\theta\), a marginal distribution:

\[ p(\theta) = \sum_{x \in \mathbf{X}} p(\theta |\mathbf{X})\ p(\mathbf{X}) \]

• Now we have the marginal distribution of \(\theta\)

• Or, we need to find the denominator for Bayes theorem to normalize our posterior distribution, a marginal distribution:

\[ p(\mathbf{X}) = \sum_{\theta \in \Theta} p(\mathbf{X} |\theta) p(\theta) \]

• We can compute \(p(\mathbf{X})\) from samples without ever directly computing the marginal

Interpreting Bayes Theorem

How can you interpret Bayes’ Theorem?

• For model parameter estimation problem:

\[Posterior\ Distribution = \frac{Likelihood \bullet Prior\ Distribution}{Evidence} \]

• Or, Bayes’ theorem in terms of model parameters:

\[ posterior\ distribution(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠│𝑑𝑎𝑡𝑎) = \\ \frac{Likelihood(𝑑𝑎𝑡𝑎|𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)\ 𝑃rior(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)}{P(data)} \]

• Summarized as:

\[ 𝑃(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠│𝑑𝑎𝑡𝑎) = \frac{P(𝑑𝑎𝑡𝑎|𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)\ 𝑃(𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠)}{P(data)} \]

Interpreting Bayes Theorem

What do these terms actually mean?

1. Posterior distribution of the parameters given the evidence or data, the goal of Bayesian analysis

2. Prior distribution is chosen to express information available about the model parameters apriori

3. Likelihood is the conditional distribution of the data given the model parameters

4. Probabiltiy of Data or evidence is the distribution of the data and normalizes the posterior

Relationships can apply to the parameters in a model; partial slopes, intercept, error distributions, lasso constants, etc

Applying Bayes Theorem

We need a tractable formulation of Bayes Theorem for computational problems

• We must avoid directly summing all of the possibilities to compute the denominator, \(P(B)\)
  • In many cases, computing this denominator directly is intractable
• Some interesting facts about conditional probabilities:

\[ 𝑃(𝐵 \cap A) = 𝑃(𝐵|𝐴)𝑃(𝐴) \\ And \\ 𝑃(𝐵)=𝑃(𝐵 \cap 𝐴)+𝑃(𝐵 \cap \bar{𝐴}) \]

Where, \(\bar{A} = not\ A\), and the marginal distribution, \(P(B)\), can be written:

\[ 𝑃(𝐵)=𝑃(𝐵|𝐴)𝑃(𝐴)+𝑃(𝐵│ \bar{𝐴})𝑃(\bar{𝐴}) \]

Applying Bayes Theorem

Using the foregoing relations we can rewrite Bayes Theorem as:

\[ P(A|B) = \frac{P(A)P(B|A)}{𝑃(𝐵│𝐴)𝑃(𝐴)+𝑃(𝐵│ \bar{𝐴})𝑃(\bar{𝐴})} \\ \]

• Computing the denominator requires summing all cases in the subsets \(A\) and \(not\ A\)
  • This is a bit of a mess!
    
  • And, the siguation is worse if there are multiple alternative hypotheses!
  • Fortunately, we can often avoid computing this denominator by force

Write Bayes Theorem as:

\[𝑃(𝐴│𝐵)=𝑘∙𝑃(𝐵|𝐴)𝑃(𝐴)\]

Ignoring the normalization constant \(k\):

\[𝑃(𝐴│𝐵) \propto 𝑃(𝐵|𝐴)𝑃(𝐴)\]

Interpreting Bayes Theorem

Denominator must account for all possible outcomes, or alternative hypotheses, \(h'\):

\[Posterior(hypothesis\ |\ evidence) =\\ \frac{Likelihood(evidence\ |\ hypothesis)\ prior(hypothesis)}{\sum_{ h' \in\ All\ possible\ hypotheses}Likelihood(evidence\ |\ h')\ prior(h')}\]

• Computing this denominator is a formidable problem!
  
• Can be infinite number of alternative hypotheses; e.g. continuous random variable

Simplified Relationship for Bayes Theorem

How to we interpret the foregoing relationship?

• Consider the following relationship:

\[Posterior\ Distribution \propto Likelihood \bullet Prior\ Distribution \\ Or\\ 𝑃( 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 │ 𝑑𝑎𝑡𝑎 ) \propto 𝑃( 𝑑𝑎𝑡𝑎 | 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 )𝑃( 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 ) \]

• We can find an un-normalized function proportional to the posterior distribution

• Sum over \(𝑃( 𝑑𝑎𝑡𝑎 | 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 )𝑃( 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 )\) to find the marginal distribution \(P(B)\)

• Approach can transform an intractable computation into a simple summation

Creating Bayes models

The goal of a Bayesian analysis is computing and performing inference on the posterior distribution of the model parameters

The general steps are as follows:

1. Identify data relevant to the research question

2. Define a sampling plan for the data. Data need not be collected in a single batch

3. Define the model and the likelihood function; e.g. regression model with Normal likelihood

4. Specify a prior distribution of the model parameters

5. Use the Bayesian inference formula to compute posterior distribution of the model parameters

6. Update the posterior as data is observed

7. Inference on the posterior can be performed; compute credible intervals

8. Optionally, simulate data values from realizations of the posterior distribution. These values are predictions from the model.

Updating Bayesian Models

An advantage of Bayesain model is that it can be updated as new observations are made

• In contrast, for frequentist models data must be collected completely in advance

• We update our belief by adding new evidence

• The posterior of a Bayesian model with no evidence is the prior

• The previous posterior serves as a prior for model updates

How can you choose a prior?

The choice of the prior is a difficult, and potentially vexing, problem when performing Bayesian analysis

• The need to choose a prior has often been cited as a reason why Bayesian models are impractical
  
• General guidance is that a prior must be convincing to a skeptical audience
  
• Often tend to use vague or less informative priors in practice

How can you choose a prior?

Some possible approaches to prior selection include:

• Use prior empirical information
  
• Apply domain knowledge to determine a reasonable distribution
  • Use information from prior work
  • Example, the viable range of parameter values could be computed from physical principles
    
  • Example, it could be well know that there is price range for some asset
• If there is poor prior knowledge for the problem a non-informative prior can be used
  • One possibility is a Uniform distribution. But be careful! since a uniform prior is informative because of limits on the values!
    
  • Other options include the Jefferys’ prior.

How can you choose a prior?

How to use prior empirical information to estimate the parameters of the prior distribution

• Deriving a prior distribution in this manner is sometimes called empirical Bayes
  • Has become more practical with large modern data sets
    
  • Method somewhere between Bayesian and frequentist
    
  • Empirical Bayes approach is often applied in practice
    
  • Some Bayesian theoreticians do not consider this a Bayesian approach at all!
• Example, need a prior distribution of home prices per square foot by location
  • Use pooled information to compute distribution of prices for all locations
    
  • Use the prior with specific evidence by locations to compute posteriors by location
    
  • Is example of hierarchical model
• Typically, a less informative prior distribution is used than the actual empirical distribution so the model is not overly constrained

Conjugate Prior Distributions

An analytically and computationally simple choice for a prior distribution family is a conjugate prior

• When a likelihood function is multiplied by its conjugate distribution the posterior distribution will be in the same family as the conjugate prior

• Attractive idea for cases where the conjugate distribution exists

  • Analytic results can be computed
    
  • The posterior is a known distribution

• But there are many practical cases where a conjugate prior is not used

  • We will address more general methods later

Conjugate Prior Distributions

Most commonly used distributions have conjugates, with a few examples:

Example using Conjugate Distribution

We are interested in analyzing the incidence of distracted drivers

• Randomly sample the behavior of 10 drivers at an intersection and determine if they exhibit distracted driving or not

• Data are Binomially distributed, a driver is distracted or not, with likelihood:

\[ P(k) = \binom{n}{k} \cdot \theta^k(1-\theta)^{n-k}\]

• Binomial likelihood has one parameter we need to estimate, \(\theta\), the probability of success

Working with Conjugate Distribution

Our process for example is:

1. Use the conjugate prior, the Beta distribution with parameters \(\alpha\) and \(\beta\) (or a,b)
   
2. Using the data sample, compute the likelihood
3. Compute the posterior distribution of distracted driving
4. Add more evidence (data) and update the posterior distribution.

Example using Conjugate Distribution

What are the properties of the Beta distribution?

Beta distribution for different parameter values

Example using Conjugate Distribution

Consider the product of a Binomial likelihood and a Beta prior

• Define the evidence as \(n\) trials with \(z\) successes

• Prior is a Beta distribution with parameters \(a\) and \(b\), or the vector \(\theta = (a,b)\)

• From Bayes Theorem the distribution of the posterior:

\[\begin{align} posterior(\theta | z, n) &= \frac{likelihood(z,n | \theta)\ prior(\theta)}{data\ distribution (z,n)} \\ p(\theta | z, n) &= \frac{Binomial(z,n | \theta)\ Beta(\theta)}{p(z,n)} \\ &= Beta(z + a,\ n-z+b) \end{align}\]

Example using Conjugate Distribution

There are some useful insights you can gain from this relationship for (discrete) integer counts:

\[ posterior(\theta | z, n) = Beta(z + a,\ n-z+b) \]

• Posterior distribution is in the Beta family, as a result of conjugacy

• Parameters \(a\) and \(b\) are determined by the prior and the evidence

• Parameters of the prior can be interpreted as pseudo counts of successes, \(a = pseudo\ success + 1\) and failures, \(b = pseudo\ failure + 1\)

  • Be careful when creating a prior to add 1 to the successes and failures
    
  • The larger the total pseudo counts, \(a + b\), the stronger the prior information

-Evidence is also in the form (actual) counts of successes, \(z\) and failure, \(n-z\)
- The more evidence the greater the influence on the posterior distribution
- Large amount of evidence will overwhelm the prior
- With large amount of evidence, posterior converges to frequentist model

Example using Conjugate Distribution

Consider example with:
- Prior pseudo counts \([1,9]\), successes \(a = 1 + 1\) and failures, \(b = 9 + 1\)
- Evidence, successes \(= 10\) and failures, \(= 30\)
- Posterior is \(Beta(10 + 2,\ 40 - 10 + 10) = Beta(12,\ 40)\)

Prior, likelihood and posterior for distracted driving

Sampling the Posterior

How can we find an estimate of the poster distribution?

1. We can sample from the analytic solution - if we have a conjugate

2. We can sample the likelihood and prior, take the product and normalize - for any posterior

3. Grid sample or Markov chain Monte Carlo (MCMC) sample

Sampling the Posterior

Grid sampling is a naive approach

• Compute the probability at each point on a regular gird
  • Sample over range of interesting values for variables
  • Posterior if conjugate prior
    
  • Prior and likelihood
• In principle can work for any number of dimensions
  • In 1-dimension is just regularly spaced points on a line
    
  • Poor scaling to higher dimensions

Sampling grid for bivariate distribution

Sampling the Posterior

Algorithm for grid sampling to compute posterior from likelihood and prior

Procedure CreateGrid(variables, lower_limits, upper_limits): 
    # Build the sampling grid 
    return sampling_grid   
    
Procedure SampleLikelihood(sampling_value, observation_values):    
    return likelihood_function(sampling_value, observation_values)
    
Procedure Prior(sampling_values, prior_parameter_value):    
    return prior_density_function(sampling_value, prior_parameter_values)    
    
ComputePosterior(variables, lower_limits, upper_limits):    
    # Initialize the sampling grid
    Grid = CreateGrid(variables, lower_limits, upper_limits)
    
    # Initialize array to hold sampled posterior values       
    array posterior[range(Grid)]

    # Compute posterior at each sampling value in the grid  
    for sampling_value in range(lower_limits, upper_limits):   
        likelihood = SampleLikelihood(sampling_value, observation_values)
        prior = Prior(sampling_values, prior_parameter_value)   
        posterior[sampling_value] = likelihood * prior

    # Normalize the posterior       
    probability_data = sum(posterior[range(Grid)])
    posterior = posterior[range(Grid)]/probability_data 
    return posterior    

Credible Intervals

How can we specify the uncertainty for a Bayesian parameter estimate?

• For frequentist analysis we use confidence intervals, but not entirely appropriate
  • Confidence intervals are based on a sampling distribution
    
  • The upper and lower confidence intervals quantiles of the sampling distribution
    
  • Bayesian analysis has no sampling distribution uses a prior distribution and likelihood
• For Bayesian analysis inference performed on posterior distribution
  • We use a concept known as the credible interval
    
  • A credible interval is an interval on the Bayesian posterior distribution with the highest \(\alpha\) proportion of posterior probability

Credible Intervals

How can we specify the uncertainty for a Bayesian parameter estimate?

• Example, the \(\alpha = 0.90\) credible interval encompasses the 90% of the posterior distribution with the highest density

• The credible interval is sometime called the highest density interval (HDI), or highest posterior density interval (HPDI)

  • These names make sense, since we seek the the densest posterior interval containing \(\alpha\) probability

• For symmetric distributions the credible interval can be numerically the same as the confidence interval

  • In general, these two quantities can be quite different

Credible Intervals

What are the 95% credible intervals for \(Beta(12,\ 40)\)?

Probability of distract drivers for next 10 cars

Credible Intervals are not Confidence Intervals

How are credible intervals different from the more familiar confidence intervals?

Confidence intervials and credible intervals are conceptually quite different

A confidence interval is a purely frequentest concept
- Is an interval on the sampling distribution where repeated samples of a statistic are expected with probability \(= \alpha\)
- Cannot interpret a confidence interval as an interval on a probability distribution of the value of a statistic!

Credible interval is an interval on a posterior distribution of the statistic
- Credible interval is exactly what the misinterpretation of the confidence interval tries to be
- Credible interval is the interval with highest \(\alpha\) probability for the statistic being estimated

For symmetric posterior distributions, the credible interval will be numerically the same as the confidence interval
- This need not be the case in general

Credible Intervals are not Confidence Intervals

Compare confidence interval and credible interval for the case of 10 observations

• Credible intervals cross the density function at exactly the same density

• Confidence intervals have the same CDF in the tails beyond the interval

Difference between credible and confidence intervals

Simulating from the posterior distribution: predictions

What else can we do with a Bayesian posterior distribution beyond credible intervals?

• Perform simulations and make predictions

• Predictions are computed by simulating from the posterior distribution

• Results of these simulations are useful for several purposes, including:

  • Predicting posterior values
    
  • Model checking by comparing simulation results agree (or not) with observations

Simulating from the posterior distribution: predictions

Example; What are the probabilities of distracted drivers for the next 10 cars with posterior, \(Beta(12,\ 40)\)?

Probability of distract drivers for next 10 cars

Summary

Bayesian analysis is a contrast to frequentist methods

• The objective of Bayesian analysis is to compute a posterior distribution
  

• Contrasts with frequentist statistics is to compute a point estimate and confidence interval from a sample

• Bayesian models allow expressing prior information in the form of a prior distribution
  

• Selection of prior distributions can be performed in a number of ways

• The posterior distribution is said to quantify our current belief
  

• We update beliefs based on additional data or evidence
  

• A critical difference with frequentist models which must be computed from a complete sample
  

• Inference can be performed on the posterior distribution by finding the maximum a postiori (MAP) value and a credible interval

• Predictions are made by simulating from the posterior distribution a

Summary

Bayesian analysis is in contrast to frequentist methods

• The objective of Bayesian analysis is to compute a posterior distribution
  

• Frequentist statistics seeks to compute a point estimate and confidence interval from a sample

• Bayesian models allow expressing prior information in the form of a prior distribution
  

• Selection of prior distributions can be performed in a number of ways

• The posterior distribution is said to quantify our current belief
  

• We update beliefs based on additional data or evidence
  

• A critical difference with frequentist models which must be computed from a complete sample

• Inference can be performed on the posterior distribution by finding the maximum a postiori (MAP) value and a credible interval